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(54) Image processing apparatus 

(57) In an image processing system, a subject ob- 
ject 210 is placed on top of an object support 220 on a 
calibration object 34 having a known pattern of features 
thereon. Images recorded at different positions and ori- 
entations are processed to generate a three-dimension- 
al computer model of the subject object alone or the sub- 
ject object together with the calibration object. By imag- 
ing the subject object 21 0 on an object support instead 
of placing it directly on the calibration object, the user is 
provided with flexibility in the selection of the pattern and 



colour of the calibration and the selection of the imaging 
positions and orientations. By providing an object sup- 
port 220 having a top surface on which the subject object 
21 0 sits which does not protrude from beneath the base 
of the subject object 21 0, the user is provided with flex- 
ibility in the selection of lighting conditions. By imaging 
the subject object 210 with a background screen 228 
behind it, the user is provided with flexibility in the se- 
lection of the surface on which the calibration object 34 
is placed for imaging. 
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Description 



[0001] The present invention relates to the recording of images of an object and the computer processing of the 
image data to determine the positions and orientations at which the images were recorded and to generate a three- 

5 dimensional (3D) computer model of the object. More particularly, the present invention relates to the use of a patterned 
calibration object for imaging together with the object, and the computer processing of the image data to determine 
the imaging positions and orientations on the basis of the calibration object's pattern in the images and to generate 
data defining a 3D computer model of the object using the calculated positions and orientations. 
[0002] 3D computer models of objects are useful for many applications. In particular, 3D computer models are often 

10 used in computer games and for computer aided design (CAD) applications. In addition, there is now a growing demand 
to have 3D computer models of objects for uses such as the embellishment of Internet sites etc. 
[0003] Many methods are known for generating 3D computer models of objects. In particular, a method is known in 
which a subject object to be modelled is placed on a flat two-dimensional calibration object having a known pattern of 
features thereon , and images of the subject object together with the calibration object are recorded at different positions 

75 and orientations. Each recorded image is then processed to calculate the position and orientation at which is was 
recorded on the basis of the positions of the features in the calibration object's pattern in the image. Subsequently, a 
3D computer model of the subject object is generated using the input images and the calculated positions and orien- . 
tations. 

[0004] Examples of this method are described, for example, in "Automatic Reconstruction of 3D Objects Using A 
20 Mobile Camera" by Niem in Image and Vision Computing 17 (1999) pages 125-134, "The Lumigraph" by Gortler et al 
in Computer Graphics Proceedings, Annual Conference Series, 1996 ACM-0-89791 -764-4/96/008, JP-A-9-170914 
and the applicant's co-pending PCT patent application PCT/GB00/04469 (WO-A-0 1/39 124) (the full contents of which 
are incorporated herein by cross-reference). 

[0005] The inventors in the present case have found that this known method suffers from a number of problems, 
25 however. 

[0006] More particularly, the inventions in the present case have found that 3D computer models generated using 
this method can be inaccurate. 

[0007] Accordingly, it is an object of the present invention to address this problem. 

[0008] The inventors of the present invention have recognised that inaccuracies in 3D computer models are caused 

30 by a number of factors as set out below. 

[0009] More particularly, shadows are often cast on the calibration object which appear to connect to the bottom of 
the subject object. Accordingly, in recorded images, the processing apparatus may not distinguish between the subject 
object and shadow, in which case the shadow is therefore determined to be part of the subject object. This problem is 
compounded because, if the subject object and calibration object are moved relative to the camera in order to record 

35 images at different positions and orientations, then the shadows move. On the other hand, if the camera itself is moved, 
the shadows cast by the camera move. 

[0010] Similarly, features in the pattern on the calibration object often touch the subject object in a two-dimensional 
recorded image and are therefore determined by the processing apparatus to be part of the subject object. 
[0011] Similarly, features on the surface on which the subject object and calibration object are placed for imaging 
40 (such as marks on a floor or table) often touch the outline of the subject object in a recorded image, and are therefore 
determined by the processing apparatus to be part of the subject object. 

[0012] Further, it is often desirable to choose the colour of the calibration object to emphasise (contrast) the pattern 
of features thereon (to assist detection in input images by the processing apparatus). However, the colour of the cali- 
bration object is often reflected onto the subject object so that the 3D computer model is generated with an incorrectly 
45 coloured surface. 

[0013] Yet further, it is difficult to record images of the subject object at low elevation angles. This is because, for 
such angles, the pattern of features on the calibration object on which the subject object is sitting is distorted in the 
images to such an extent that detection of the features in input images by the processing apparatus in a reliable manner 
is not possible. By not being able to record images from low elevation angles, input images showing in detail parts of 
so the subject object may not be available, resulting in those parts being inaccurately reproduced in the 3D computer 
model. 

[0014] Accordingly, it is an object of the present invention to address one or more of these problems. 
[0015] In terms of solutions, the inventors have recognised that the problem of shadows could be addressed by using 
diffuse illumination which is symmetrical about the subject object and calibration object. However, this is very difficult 
55 and/or expensive to achieve in practice. 

[0016] Accordingly, it is an object of the present invention to provide a different solution. 

[0017] According to the present invention, there is provided a method of recording and processing images of a subject 
object to generate a three-dimensional computer model, in which the subject object is held in a fixed configuration 
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relative to a calibration object by an object support while images are recorded at different camera positions and orien- 
tations, the image data is processed to determine the imaging positions and orientations on the basis of features from 
the calibration object visible in the images, and a three-dimensional computer model of the subject object alone or the 
subject object together with the object support is generated using the calculated positions and orientations. 
5 [0018] The present invention also provides apparatus components for carrying out the above method, including a 
processing apparatus and a computer program, and methods relating thereto. 

[001 9] By imaging the subject object away from the surface of the calibration object (for example on an object stand) 
instead of placing the subject object directly on the calibration object, the user is provided with greater flexibility in the 
selection of the pattern on the calibration object, the selection of the colour of the calibration object and the selection 
10 of imaging positions and orientations. This is because the problem of features from the pattern on the calibration object 
appearing to touch the subject object in input images, the problem of the calibration object colour being reflected onto 
the subject object and the problem of recording input images form low elevation angles are ail addressed by moving 
the subject object away from the calibration object. 

[0020] Preferably, the top surface of the object support on which the subject object is placed is shaped and sized so 
15 that there is no horizontal surface of any substantial size protruding from beneath the base of the subject object when 
the subject object is placed on the object support. In this way, the user is provided with greater flexibility in the selection 
of lighting conditions because the problem of shadows appearing as part of the subject object in input images is also 
addressed. 

[0021] Preferably, the subject object is imaged with a screen behind it. In this way, the user is provided with greater 
20 flexibility in the selection of the surface on which the calibration object, object support and subject object are placed 
for imaging because the problem of marks on the surface appearing as part of the subject object in input images is 
addressed. 

[0022] Embodiments of the invention will now be described, by way of example only, with reference to the accom- 
panying drawings, in which: 

25 

Figure 1 schematically shows the components of an embodiment of the invention, together with the notional func- 
tional processing units into which the processing apparatus component can be thought of as being configured 
when programmed by programming instructions; 

30 Figure 2 illustrates the connection of a photographic mat to a mirror; 

Figure 3 illustrates the recording of images of a subject object for which a 3D computer model is to be generated, 
together with an object support and photographic mat; 

35 Figures 4a-4e illustrate how the choice of camera viewing position and orientation and the choice of height of 

object support affect the input images recorded by the camera; 

Figure 5 illustrates images of the object, object support and photographic mat which are input to the processing 
apparatus in Figure 1 ; 

40 

Figure 6 shows the processing operations performed by the processing apparatus in Figure 1 to process input data; 

Figure 7 shows the processing operations performed at step S6-8 in Figure 6; 

45 Figure 8 illustrates the definition of an initial volume at step S6-10 in Figure 6 on which to perform voxel carving 

operations to generate the 3D surface model of the subject object and object support; 

Figure 9 shows the processing operations performed at step S6-12 in Figure 6; 

so Figure 10 illustrates the creation of pixel data for a virtual image at step S9-2 in Figure 9; 

Figure 11 shows the processing operations performed in the first embodiment at step S6-14 in Figure 6; 

Figures 12a, 12b and 12c illustrate the display and movement of the moveable plane at steps S11-2 and S11-4 in 
55 Figure 11; 

Figure 1 3 illustrates the re-definition of an initial volume on which to perform voxel carving at step S1 1 -8 in Figure 1 1 ; 
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Figure 14 illustrates an object support having a transparent top portion in a second embodiment; 

Figure 15 shows the processing operations performed in the second embodiment at step S6-14 in Figure 6; 

5 Figure 1 6 shows an object support having a marker thereon to assist the user in selecting camera imaging positions 

and orientations in a further embodiment; 

Figure 1 7 illustrates an object support in a further embodiment, in which identifiable features are provided on the 
object support rather than on a photographic mat; 

10 

Figure 1 8 illustrates a photographic mat and object support in a further embodiment, in which identifiable features 
are provided on both the photographic mat and object support; 

Figure 19 illustrates a further embodiment, in which the photographic mat is replaced by a three-dimensional 
15 calibration object having identifiable features thereon; and 

Figures 20a and 20b illustrate how a three-dimensional computer model may be edited using a plane of variable 
position and orientation in all embodiments. 

20 First Embodiment 

[0023] Referring to Figure 1 , an embodiment of the invention comprises a processing apparatus 2, such as a persona! 
computer, containing, in a conventional manner, one or more processors, memories, graphics cards etc : together with 
a display device 4, such as a conventional personal computer monitor, user input devices 6, such as a keyboard, mouse 
25 etc, a printer 8, and a display panel 1 0 comprising a flat panel having controllable pixels, such as the PL400 manufac- 
tured by WACOM. 

[0024] The processing apparatus 2 is programmed to operate in accordance with programming instructions input, 
for example, as data stored on a data storage medium, such as disk 12, and/or as a signal 14 input to the processing 
apparatus 2, for example form a remote database, by transmission over a communication network (not shown) such 
30 as the Internet or by transmission through the atmosphere, and/or entered by a user via a user input device 6 such as 
a keyboard. 

[0025] As will be described in more detail below, the programming instructions comprise instructions to cause the 
processing apparatus 2 to become configured to process input data defining a plurality of images of one or more subject 
objects recorded at different positions and orientations to calculate the positions and orientations at which the input 

35 images were recorded and to use the calculated positions and orientations to generate data defining a three-dimen- 
sional computer model of the subject object(s). In this embodiment, the subject object is placed on an object support 
and the subject object and support are imaged on a calibration object (a two-dimensional photographic mat in this 
embodiment) which has a known pattern of features thereon. The positions and orientations at which the input images 
were recorded are calculated by detecting the positions of the features of the calibration object pattern in the images. 

40 As will be explained in more detail below, by using an object support to image the subject object, accurate three- 
dimensional computer models of the subject object can be consistently obtained. 

[0026] When programmed by the programming instructions, processing apparatus 2 can be thought of as being 
configured as a number of functional units for performing processing operations. Examples of such functional units 
and their interconnections are shown in Figure 1 . The units and interconnections illustrated in Figure 1 are, however, 

45 notional and are shown for illustration purposes only to assist understanding; they do not necessarily represent units 
and connections into which the processor memory etc of the processing apparatus 2 become configured. 
[0027] Referring to the functional units shown in Figure 1 , a central controller 20 processes inputs from the user input 
devices 6, and also provides control and processing for the other functional units. Memory 24 is provided for use by 
central controller 20 and the other functional units. 

so [0028] Mat generator 30 generates control signals to control printer 8 or display panel 1 0 to print a photographic mat 
34 on a recording medium such as a piece of paper, or to display the photographic mat on display panel 1 0. As will be 
described in more detail below, the photographic mat comprises a predetermined pattern of features and the subject 
object(s) for which a three-dimensional computer model is to be generated is placed on an object support on the printed 
photographic mat 34 or on the display panel 10 on which the photographic mat is displayed. Images of the subject 

55 object(s), object support and the photographic mat are then recorded and input to the processing apparatus 2. 

[0029] The colour of the recording medium on which printer 8 prints the photographic mat is chosen to maximise the 
contrast between the recording medium and the features in the pattern of the photographic mat (thereby increasing 
the likelihood that the features will be located accurately in the input images by processing apparatus 2). The colour 
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can be freely chosen in this embodiment without having an adverse effect on the 3D computer model subsequently 
generated because the effect of light reflected from the printed photographic mat onto the subject object is minimised 
by placing the subject object on an object support and thereby raising the subject object away from the photographic 
mat. It has been found in practice that light, unsaturated colours give good results when chosen as the colour of the 
5 recording medium. 

[0030] Mat generator 30 stores data defining the pattern of features printed or displayed on the photographic mat 
for use by the processing apparatus 2 in calculating the positions and orientations at which the input images were 
recorded. More particularly, mat generator 30 stores data defining the pattern of features together with a coordinate 
system relative to the pattern of features (which, in effect, defines a reference position and orientation of the photo- 

10 graphic mat), and processing apparatus 2 calculates the positions and orientations at which the input images were 
recorded in the defined coordinate system (and thus relative to the reference position and orientation). 
[0031] In this embodiment, the pattern on the photographic mat comprises spatial clusters of features for example 
as described in co-pending PCT patent application PCT/G BOO/04469 (WO-A-0 1/391 24) (the full contents of which are 
incorporated herein by cross-reference) or any known pattern of features, such as a pattern of coloured dots, with each 

15 dot having a different hue/brightness combination so that each respective dot is unique, for example as described in 
JP-A-9-170914, a pattern of concentric circles connected by radial line segments with known dimensions and position 
markers in each quadrant, for example as described in "Automatic Reconstruction of 3D Objects Using A Mobile Cam- 
era" by Niem in Image and Vision Computing 17 (1999) pages 125-134, or a pattern comprising concentric rings with 
different diameters, for example as described in 'The Lumigraph" by Gortler et al in Computer Graphics Proceedings, 

20 Annual Conference Series, 1 996 ACM-0-89791 -764-4/96/008. 

[0032] In the remainder of the description of this embodiment, it will be assumed that the pattern is printed by printer 
8 on a recording medium (in this embodiment, a sheet of paper) to generate a printed photographic mat 34, although, 
as mentioned above, the pattern could be displayed on display panel 1 0 instead. 

[0033] Input data store 40 stores input data input to the processing apparatus 2 for example as data stored on a 
25 storage device, such as disk 42, as a signal 44 transmitted to the processing apparatus 2, or using a user input device 
6. The input data defines a plurality of images recorded at different positions and orientations showing one or more 
subject objects and an object support on the photographic mat. In this embodiment, the input data also defines an 
input image showing the background against which the subject object(s) and object support were imaged, together 
with part of the photographic mat to show the background colour thereof or a different object having the same colour 
30 as the background colour of the mat. In addition, in this embodiment, the input data also includes data defining the 
intrinsic parameters of the camera which recorded the images, that is, the aspect ratio, focal length, principal point (the 
point at which the optical axis intersects the imaging plane), first order radial distortion coefficient, and skew angle (the 
angle between the axes of the pixel grid; because the axes may not be exactly orthogonal). 

[0034] The input data defining the input images may be generated for example by downloading pixel data from a 
35 digital camera which recorded the images, or by scanning photographs using a scanner (not shown). The input data 
defining the intrinsic camera parameters may be input by a user using a user input device 6. 

[0035] Camera calculator 50 processes each input image to detect the positions in the image of the features on the 
photographic mat and to calculate the position and orientation of the camera when the input image was recorded. 
[0036] Image data segmenter 60 processes each input image to separate image data corresponding to the subject 
40 object and object support from other image data in the image. 

[0037] Surface modeller 70 processes the segmented image data produced by image data segmenter 60 and the 
data defining the positions and orientations at which the images were recorded generated by camera calculator 50, to 
generate data defining a 3D computer model representing the actual surfaces of the subject object(s) and object support 
in the input images. 

45 [0038] Surface texturer 80 generates texture data from the input image data for rendering onto the surface model 
produced by surface modeller 70. 

[0039] Object/support separator 90 performs processing to amend the three-dimensional computer model of the 
subject object and object support generated by surface modeller 70 to delete the part of the model corresponding to 
the object support, so as to leave a three-dimensional computer model of the subject object alone. 
so [0040] Display processor 110, under the control of central controller 20, displays instructions to a user via display 
device 4. In addition, under the control of central controller 20, display processor 110 also displays images of the 3D 
computer model form a user-selected viewpoint by processing the surface model data generated by surface modeller 
70 and/or object/support separator 90 and rendering texture data produced by surface texturer 80 onto the surface 
model. 

55 [0041] Output data store 120 stores the camera positions and orientations calculated by camera calculator 50 for 
each input image, the image data relating to the subject object and object support from each input image generated 
by image data segmenter 60, and also the surface model and texture data therefor generated by surface modeller 70, 
object/support separator 90, and surface texturer 80. Central controller 20 controls the output of data from output data 
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store 120, for example as data on a storage device, such as disk 122, and/or as a signal 124. 

[0042] Referring to Figure 2, in this embodiment the pattern of features on the photographic mat 34 is printed by 
printer 8 around a central blank area which is cut out to leave a hole 130 in the photographic mat 34. 
[0043] The photographic mat 34 with the hole 130 therein is then attached to a mirror 140, for example by gluing. 
5 As will be explained in more detail below, hole 130 and mirror 140 enable the reflection of the bottom surface of the 
subject object to be seen in input images : and hence texture data for the bottom surface of the subject object in the 
three-dimensional computer model can be generated. 

[0044] Referring to Figure 3, the mirror 1 40 with the printed photographic mat 34 attached thereto is placed on a flat 
surface 200, and the subject object 210 for which a 3D computer model is to be generated is placed on an object 
10 support 220 on the photographic mat 34 so that the object support 220 is positioned in the hole 1 30 and is surrounded 
by the features making up the pattern on the mat. 

[0045] By raising subject object 210 off the photographic mat 34 using object support 220, any shadows on the 
photographic mat 34 will touch the object support 220 and not the subject object 210. Similarly, features in the pattern 
on the photographic mat will appear to touch only the object support 220 and not the subject object 210 in recorded 
15 images. Accordingly in the 3D computer model generated by processing apparatus 2, errors may occur in the model 
of the object support 220 but the subject object 210 will be accurately modelled. However, the errors in the object 
support 220 are unimportant and, in this embodiment, object/support separator 90 removes the 3D computer model 
of the object support 220 leaving a 3D computer model of the subject object 210 on its own. 

[0046] Yet further, as will be explained below, by raising the subject object 210 above the photographic mat 34, 
20 images at low elevation angles can be recorded in which the pattern of features on the photographic mat 34 is not 
distorted to such an extent that processing apparatus 2 cannot perform processing to identify the features in the input 
images and calculate the imaging positions and orientations. Accordingly, the accuracy of the 3D computer model of 
the subject object 21 0 is further increased because images showing the necessary detail of the subject object 21 0 can 
be recorded and processed. 

25 [0047] In this embodiment, the object support 220 comprises a horizontal plate 222 of a transparent material, such 
as glass, supported by four cylindrical opaque legs 224. Accordingly, because the horizontal plate 222 is transparent, 
the bottom surface of the subject object 210 is visible as a reflection in the mirror 140. 

[0048] A background screen 228 of a substantially uniform colour is placed behind the subject object 21 0 and object 
support 220. In this way, no part of the subject object 210 (or object support 220) appears against the surface 200 in 

30 recorded images. Instead, the subject object 210 appears only against the background screen. This allows the user 
to choose freely the surface 200 on which the photographic mat 34 is placed, because marks on the surface 200 or 
regions of non-uniform colour on the surface 200, etc. will not touch the outline of the subject object 21 0 in any input 
image. Accordingly, as will be explained in more detail below, the use of background screen 228 assists in separating 
image data relating to the subject object 210 and object support 220 from other image data during segmentation 

35 processing by image data segmenter 60. 

[0049] Images of the subject object 21 0, object support 220 and photographic mat 34 are recorded at different po- 
sitions and orientations to show different parts of subject object 210 using a digital camera 230. In this embodiment, 
data defining the images recorded by camera 230 is input to processing apparatus 2 as a signal 44 along wire 232. 
[0050] More particularly, in this embodiment, camera 230 and background screen 228 remain in fixed positions and 

40 photographic mat 34 with the subject object 21 0 and object support 220 thereon is moved (translated) and rotated (for 
example in the direction of arrow 240) on surface 200, and photographs of the subject object 21 0 at different positions 
and orientations relative to the camera 230 are recorded. During the rotation and translation of the photographic mat 
34 on surface 200, the subject object 210 and object support 220 do not move relative to the mat 34. 
[0051] Figures 4a to 4e illustrate how, in this embodiment, the location and viewing angle of camera 230 are chosen 

45 and how the height of the object support 220 is chosen. 

[0052] Referring to Figure 4a, if camera 230 is located in position (i), that is, a highly elevated position looking down 
on the subject object 210, then, as shown in Figure 4b, in each image recorded by camera 230, the top part of the 
subject object 210 will be outlined against the background screen 228, but the bottom part of the subject object 210 
will be outlined against the photographic mat 34 and/or the horizontal plate 222 of the object support 220. This is 

50 undesirable because any shadows on the horizontal plate 222 and photographic mat 34 may appear as part of the 
subject object 21 0 and hence may not be separated from the subject object 21 0 by image data segmenter 60. Similarly, 
one or more features in the pattern on the photographic mat 34 may touch the subject object 210 in an input image 
(and hence appear to be part of the subject object 21 0), and may not therefore be separated from the subject object 
210 by image data segmenter 60. 

55 [0053] Accordingly, a more desirable imaging location and viewing direction is that marked (ii) in Figure 4a, in which 
the optical axis of the camera 230 is substantially in line with the horizontal plate 222 of the object stand 220. However, 
if the height of the horizontal plate 222 above photographic mat 34 is too low (as is the case in the example shown in 
Figure 4a), then, as shown in Figure 4c, in each input image recorded by camera 230, the photographic mat 34 will 
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appear distorted to such an extent that the features in the pattern thereon can not be distinguished from each other. 
[0054] Consequently, as shown in Figure 4d, the position and viewing direction of camera 230 are chosen so that 
the optical axis of the camera is substantially in line with the horizontal plate 222 of the object support 220, and the 
height of the object support 220 is chosen so that the pattern of features on the photographic mat 34 is not distorted 

5 in the input images. In this way, referring to Figure 4e, which shows an example of an input image recorded using the 
configuration shown in Figure 4d, the subject object 21 0 is surrounded by the background screen 228 (thereby enabling 
image data segmenter 60 to segment image data relating to the subject object 21 0 more accurately) and the features 
in the pattern on the photographic mat 34 can be distinguished from each other (thereby enabling camera calculator 
50 to detect the position of each feature in an input image and hence determine the position and orientation at which 

10 the input image was recorded). It should be noted that the background screen 228 does not need to extended to the 
edges of the input images (as is the case in the example shown in Figure 4e), but only needs to be extended above, 
below and to the sides of the subject object 21 0 so that the subject object 21 0 is surrounded thereby. 
[0055] In summary, by raising the subject object 210 off the photographic mat 34 (this being achieved in this embod- 
iment using object support 220) input images can be recorded in which shadows on the photographic mat 34 and 

15 features in the pattern on photographic mat 34 do not touch the outline of the subject object 210. Further, the colour 
of the photographic mat 34 can be freely chosen to emphasise the features in the pattern thereon because the colour 
is reflected much less onto the subject object 210. Further, images of the subject object 210 can be recorded at low 
elevation angles, ensuring that suitable images are available to model all parts of the subject object 21 0 accurately in 
the 3D computer model. 

20 [0056] In addition, by placing background screen 228 behind the subject object 210, the choice of surface 200 can 
be freely made because input images can be recorded in which no part of the surface 200 is seen to touch the subject 
object 210. 

[0057] Figure 5 shows examples of input images 300, 302, 304 and 306 of the subject object 210, object support 
220 and photographic mat 34 in different positions and orientations relative to camera 230. 

25 [0058] In this embodiment, following the recording and input of images of subject object 21 0, object support 220 and 
photographic mat 34, a further image is recorded and input to processing apparatus 2. This further image comprises 
a "background image", which is an image of the surface 200, background screen 228 and an object having the same 
colour as the paper on which photographic mat 34 is printed. Such a background image may be recorded by placing 
a blank sheet of paper having the same colour as the sheet on which photographic mat 34 is recorded on surface 200 

30 in front of the background screen 228, or by turning the photographic mat 34 over on surface 200 so that the pattern 
thereon is not visible in the image. 

[0059] Figure 6 shows the processing operations performed by processing apparatus 2 to process input data in this 
embodiment. 

[0060] Referring to Figure 6, at step S6-2, central controller 20 causes display processor 11 0 to display a message 

35 on display device 4 requesting the user to input data for processing. 

[0061] At step S6-4, data input by the user in response to the request at step S6-2 is stored in the input data store 
40. More particularly, in this embodiment, the input data comprises image data defining the images of the subject object 
210, object support 220 and mat 34 recorded at different positions and orientations relative to the camera 230, the 
"background image" showing the surface 200 on which photographic mat 34 was placed to record the input images 

40 together with the background screen 228 and an object having the same colour as the recording material on which the 
pattern of photographic mat 34 is printed, and data defining the intrinsic parameters of the camera 230 which recorded 
the input images, that is the aspect ratio, focal length, principal point (the point at which the optical axis intersects the 
imaging plane), the first order radial distortion coefficient, and the skew angle (the angle between the axes of the pixel 
grid). 

45 [0062] At step S6-6, camera calculator 50 processes the input data stored at step S6-4 to determine the position 
and orientation of the camera 230 relative to the photographic mat 34 (and hence relative to the subject object 210 
and object support 220) for each input image. This processing comprises, for each input image, detecting the features 
in the image which make up the pattern on the photographic mat 34 and comparing the features to the stored pattern 
for the photographic mat to determine the position and orientation of the camera 230 relative to the mat. The processing 

50 performed by camera calculator 50 at step S6-6 depends upon the pattern of features used on the photographic mat 
34. Accordingly, suitable processing is described, for example, in co-pending PCT patent application PCT/GB00/04469 
(WO-A-0 1/39 124), JP-A-9-170914, "Automatic Reconstruction of 3D Objects Using A Mobile Camera" by Niem in 
Image and Vision Computing 17 (1999) pages 125-134 and "The Lumigraph" by Gortler et al in Computer Graphics 
Proceedings, Annual Conference Series, 1996 AC M-0-89791 -764-4/96/008. 

55 [0063] At step S6-8, image data segmenter 60 processes each input image to segment image data representing the 
subject object 21 0 and object support 220 from image data representing the photographic mat 34, the surface 200 on 
which the mat 34 is placed and the background screen 228 (step S6-8 being a preliminary step in this embodiment to 
generate data for use in the subsequent generation of a 3D computer model of the surface of subject object 21 0 and 
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object support 220, as will be described in more detail below). 

[0064] Figure 7 shows the processing operations performed by image data segmenter 60 at step S6-8. 
[0065] Referring to Figure 7, at steps S7-2 to S7-10, image data segmenter 60 builds a hash table of quantised 
values representing the colours in the input images which represent the photographic mat 34, the surface 200 and the 
background screen 228 but not the subject object 21 0 and object support 220. 

[0066] More particularly, at step S7-2, image data segmenter 60 reads the RBG data values for the next pixel in the 
"background image" stored at step S6-4 in Figure 6 (that is, the final image to be input to processing apparatus 2 which 
shows the surface 200, the background screen 228 and an object having the same colour as the material on which 
photographic mat 34 is printed). 

[0067] At step S7-4, image data segmenter 60 calculates a quantised red (R) value, a quantised green (G) and a 
quantised blue (B) value for the pixel in accordance with the following equation: 

where: 

"q" is the quantised value; 

"p" is the R, G or B value read at step S7-2; 

T is a threshold value determining how near RGB values from an input image showing the subject object 210 
need to be to background colours to be labelled as background. In this embodiment, T is set to 4. 

[0068] At step S7-6, image data segmenter 60 combines the quantised R, G and B values calculated at step S7-4 
into a "triple value" in a conventional manner. 

[0069] At step S7-8, image data segmenter 60 applies a hashing function to the quantised R, G and B values cal- 
culated at step S7-4 to define a bin in a hash table, and adds the "triple" value defined at step S7-6 to the defined bin. 
More particularly, in this embodiment, image data segmenter 60 applies the following hashing function to the quantised 
R, G and B values to define the bin in the hash table: 



h(q)= (q red & 7T 2*6+ (q green &7y 2*3+ (q bfue &7) (2) 

[0070] That is, the bin in the hash table is defined by the three least significant bits of each colour. This function is 
chosen to try and spread out the data into the available bins in the hash table, so that each bin has only a small number 
of "triple" values. In this embodiment, at step S7-8, the "triple" value is added to the bin only if it does not already exist 
therein, so that each "triple" value is added only once to the hash table. 

[0071] At step S7-10, image data segmenter 60 determines whether there is another pixel in the background image. 
Steps S7-2 to S7-10 are repeated until each pixel in the "background" image has been processed in the manner 
described above. As a result of this processing, a hash table is generated containing values representing the colours 
in the "background" image. 

[0072] At steps S7-12 to S7-48, image data segmenter 60 considers each input image in turn and uses the hash 
table to segment the data in the input image relating to the photographic mat 34, the background screen 228 and the 
surface 200 from the data in the input image relating to the subject object 21 0 and object support 220. 
[0073] In this embodiment, the "background" image processed at steps S7-2 to S7-10 to generate the hash table 
does not show the features on the photographic mat 34. Accordingly, the segmentation performed at steps S7-12 to 
S7-46 does not distinguish pixel data relating to the subject object 21 0 and object support 220 from pixel data relating 
to a feature on the mat 34. Instead, in this embodiment, the processing performed by surface modeller 70 to generate 
the 3D computer model of the surface of subject object 21 0 and object support 220 is carried out in such a way that 
pixels relating to a feature on photographic mat 34 do not contribute to the surface model, as will be described in more 
detail below. 

[0074] At step S7-12, image data segmenter 60 considers the next input image, and at step S7-14 reads the R, G 
and B values for the next pixel in the input image (this being the first pixel the first time step S7-14 is performed). 
[0075] At step S7-1 6, image data segmenter 60 calculates a quantised R value, a quantised G value and a quantised 
B value for the pixel using equation (1) above. 

[0076] At step S7-1 8, image data segmenter 60 combines the quantised R, G and B values calculated at step S7-16 
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into a "triple value". 

[0077] At step S7-20, image data segmenter 60 applies a hashing function in accordance with equation (2) above 
to the quantised values calculated at step S7-16 to define a bin in the hash table generated at steps S7-2 to S7-1 0. 
[0078] At step S7-22, image data segmenter 60 reads the "triple" values in the hash table bin defined at step S7-20, 
5 these "triple" values representing the colours of the material of the photographic mat 34, the background screen 228 
and the surface 200. 

[0079] At step S7-24, image data segmenter 60 determines whether the "triple" value generated at step S7-1 8 of 
the pixel in the input image currently being considered is the same as any of the background "triple" values in the hash 
table bin. 

10 [0080] If it is determined at step S7-24 that the "triple" value of the pixel is the same as a background "triple" value, 
then, at step S7-26, it is determined that the pixel is a background pixel and the value of the pixel is set to "black". 
[0081] On the other hand, if it is determined at step S7-24 that the "triple" value of the pixel is not the same as any 
"triple" value of the background, then, at step S7-28, it is determined that the pixel is part of the subject object 21 0 or 
object support 220 and image data segmenter 60 sets the value of the pixel to "white". 

15 [0082] At step S7-30, image data segmenter 60 determines whether there is another pixel in the input image. Steps 
S7-14 to S7-30 are repeated until each pixel in the input image has been processed in the manner described above. 
[0083] At steps S7-32 to S7-44, image data segmenter 60 performs processing to correct any errors in the classifi- 
cation of image pixels as background pixels or object pixels. 

[0084] More particularly, at step S7-32, image data segmenter 60 defines a circular mask for use as a median filter. 
20 in this embodiment, the circular mask has a radius of 4 pixels. 

[0085] At step S7-34, image data segmenter 60 performs processing to place the centre of the mask defined at step 
S7-32 at the centre of the next pixel in the binary image generated at steps S7-26 and S7-28 (this being the first pixel 
the first time step S7-34 is performed). 

[0086] At step S7-36, image data segmenter 60 counts the number of black pixels and the number of white pixels 
25 within the mask. 

[0087] At step S7-38, image data segmenter 60 determines whether the number of white pixels within the mask is 
greater than or equal to the number of black pixels within the mask. 

[0088] If it is determined at step £7-38 that the number of white pixels is greater than or equal to the number of black 

pixels, then, at step S7-40 image data segmenter 60 sets the value of the pixel on which the mask is centred to white. 
30 On the other hand, if it is determined at step S7-38 that the number of black pixels is greater than the number of white 

pixels then, at step S7-42, image data segmenter 60 sets the value of the pixel on which the mask is centred to black. 

[0089] At step S7-44, image data segmenter 60 determines whether there is another pixel in the binary image, and 

steps S7-34 to S7-44 are repeated until each pixel has been processed in the manner described above. 

[0090] At step S7-48, image data segmenter 60 determines whether there is another input image to be processed. 
35 Steps S7-12 to S7-48 are repeated until each input image has been processed in the manner described above. 

[0091] Referring again to Figure 6, at step S6-1 0, surface modeller 70 performs processing to generate data defining 

a 3D computer model of the surface of subject object 210 and the surface of object support 220. 

[0092] In this embodiment, the processing at step S6-1 0 is performed in a conventional manner, and comprises the 

following three stages: 

40 

(1) The camera positions and orientations generated at step S6-6 and the segmented image data generated at 
step S6-8 is processed to generate a voxel carving, which comprises data defining the 3D grid of voxels enclosing 
the surface of subject object 210 and the surface of object support 220. Surface modeller 70 performs processing 
for this stage in a conventional manner, for example as described in "Rapid Octree Construction from Image Se- 

45 quences" by R. Szeliski in CVGIP: Image Understanding, Volume 58, Number 1 , July 1 993, pages 23-32. However, 

referring to Figure 8, in this embodiment, the start volume defined by surface modeller 70 on which to perform the 
voxel carve processing comprises a cuboid 400 having vertical side faces 402, 404, 406, 408, a horizontal top 
face 410 and a horizontal bottom face 412. The vertical side faces 402, 404, 406, 408 are positioned so that they 
touch the edge of the pattern of features on the photographic mat 34 (as schematically represented by the outer 

50 dotted circle in Figure 8). Accordingly, the vertical side faces wholly contain the subject object 210. The position 

of the top face 410 is defined by intersecting a line 420 from the focal point of the camera 230 through the top 
edge of any one of the input images stored at step S6-4 with a vertical line 422 through thecentre of the photographic 
mat 34. More particularly, the focal point of the camera 230 and the top edge of an image are known as a result 
of the position and orientation calculations performed at step S6-6 and, by setting the height of the top face 410 

55 to correspond to the point where the line 420 intersects a vertical line 422 through the centre of the photographic 

mat 34, the top face 41 0 will always be above the top of the subject object 21 0 (provided that the top of the subject 
object 21 0 is visible in each input image). The position of the horizontal base face 41 2 is set to be slightly above 
the plane of the photographic mat 34. By setting the position of the base face 41 2 in this way, features in the pattern 
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on the photographic mat 34 (which were not separated from the subject object 210 and object support 220 in the 
image segmentation processing performed at step S6-8) will be disregarded during the voxel carving processing 
and a 3D surface model of the subject object 210 and object support 220 will be generated. 

5 (2) The data defining the voxel carving is processed to generate data defining a 3D surface mesh of triangles 

defining the surface of the subject object 210 and the surface of the object support 220. In this embodiment, this 
stage of the processing is performed by surface modeller 70 in accordance with a conventional marching cubes 
algorithm, for example as described in W.E.Lorensen and H.E.CIine: "Marching Cubes: A High Resolution 3D 
Surface Construction Algorithm", in Computer Graphics, SIGGRAPH 87 proceedings, 21 : 163-169, July 1987, or 

10 J. Bloomenthal: "An Implicit Surface Polygonizer", Graphics Gems IV, AP Professional, 1994, ISBN 0123361559, 

pp 324-350. 

(3) The number of triangles in the surface mesh generated at stage 2 is substantially reduced by performing a 
decimation process. 

15 

[0093] In stage 3, surface modeller 70 performs processing in this embodiment to carry out the decimation process 
by randomly removing vertices from the triangular mesh generated in stage 2 to see whether or not each vertex con- 
tributes to the shape of the surface of subject object 21 0 or the surface of object support 220. Vertices which do not 
contribute to the shape are discarded from the triangulation, resulting in fewer vertices (and hence fewer triangles) in 
20 the final model. The selection of vertices to remove and test is carried out in a random order in order to avoid the effect 
of gradually eroding a large part of the surface by consecutively removing neighbouring vertices. The decimation al- 
gorithm performed by surface modeller 70 in this embodiment is described below in pseudo-code. 
INPUT 

Read in vertices 
25 Read in triples of vertex IDS making up triangles 
PROCESSING 
Repeat NVERTEX times 

Choose a random vertex V 7 which hasn't been chosen before 

Locate set of all triangles having V as a vertex, S Order S so adjacent triangles are next to each other Re-trian- 
30 gulate triangle set, ignoring V (i.e. remove selected triangles & V and then fill in hole) 
Find the maximum distance between V and the plane of each triangle 
If (distance < threshold) 

Discard V and keep new triangulation 

Else 

35 Keep V and return to old triangulation 

OUPUT 

Output list of kept vertices 
Output updated list of triangles 

[0094] Since the absolute positions of the features on photographic mat 34 are known (the features having been 
40 printed in accordance with prestored data defining the positions), the 3D computer model of the surface of subject 
object 21 0 and the surface of object support 220 is generated at step S6-1 0 to the correct scale. 
[0095] At step S6-12, surface texturer 80 processes the input image data stored at step S6-4 to generate texture 
data for each triangle in the surface model generated by surface modeller 70 at step S6-10. 

[0096] Figure 9 shows the processing operations performed by surface texturer 90 at step S6-1 2 in this embodiment. 

45 [0097] Referring to Figure 9, at step S9-2, surface texturer 90 generates image data which can be used as texture 
data for the bottom surface of the subject object 210, that is the surface which sits on the horizontal plate 222 of the 
object support 220. In this embodiment, this processing is performed by converting each input image of the subject 
object 210 stored at step S6-4 into a virtual image having a camera viewing position and orientation such that the 
underside of the subject object 21 0 can be seen and comprising the pixel data from the input image which corresponds 

so to the reflection of the underside of the subject object 21 0 in the mirror 1 40. More particularly, as noted previously, mat 
generator 30 stores data defining a coordinate system and camera calculator 50 calculates the position and orientation 
at which each input image was recorded in the defined coordinate system. In this embodiment, the centre of the pho- 
tographic mat 34 is the origin of the coordinate system and the z-direction is the direction perpendicular to the plane 
of the photographic mat, pointing up. At step S9-2, surface texturer 80 processes each input image by changing the 

55 position and orientation of the input image calculated by camera calculator 50 to define a new position and orientation 
for the virtual image corresponding to a position and orientation which shows the underside of a subject object 210. 
More particularly, representing the position and orientation of the input image calculated by camera calculator 50 as a 
3 by 4 matrix M, surface texturer 80 generates a 3 by 4 matrix M' defining the position and orientation of the virtual 
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M = M 
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[0098] Surface texturer 80 generates pixel data for each virtual image by masking out all pixels in the corresponding 
15 input image which do not correspond to the hole 1 30 in the photographic mat 34 (the position of hole 130 in the input 
image being known from the position and orientation calculations performed by camera calculator 50 at step S6-6). In 
this way, referring to Figure 10, the pixel data for a virtual image comprises pixel data showing the reflection in the 
mirror 140 with all of the remaining pixel data masked out (indicated by the shading in Figure 1 0). 
[0099] It will be appreciated that, in some input images, the legs 224 of the object support 220 will appear in the 
20 reflection in mirror 1 40 and hence will appear in the corresponding virtual image. However, this is not a problem in this 
embodiment since, as will be clear from the description below, the texture data for the different triangles in the 3D 
computer model making up the bottom surface of subject object 210 does not necessarily come from the same input 
image, and therefore the effect of the legs 224 being visible in some of the input images will not be significantly no- 
ticeable. Alternatively, the legs 24 may be made of a transparent material, so that they are not visible in the reflection 
25 from mirror 140. 

[01 00] Referring again to Figure 9, at steps S9-4 to S9-1 0, surface texturer 80 performs processing in a conventional 
manner to select each triangle in the surface mesh generated at step S6-1 0 and to define texture data from one of the 
input images for a selected triangle. 

[0101] More particularly, at step S9-4, surface texturer 80 considers the next triangle in the surface mesh generated 
30 at step S6-1 0 (this being the first triangle the first time step S9-4 is performed) and, at step S9-6, performs processing 
to find the input image "i" which is most front-facing to the triangle. 

[0102] More particularly, at step S9-6, surface texturer 80 finds the input image for which the value n t .Vj is largest, 
where n t is the triangle normal and Oj is the viewing direction for the "i"th image. This identifies the input image in which 
the surface triangle has the largest projected area. 

35 [0103] At step S9-8, surface texturer 80 projects the triangle into the input image identified at step S9-6, and stores 
the vertices of the projected triangle as texture coordinates defining an image texture map in a conventional manner 
[0104] At step S9-1 0, surface texturer 80 determines whether there is anothertriangle in the surface mesh, and steps 
S9-4 to S9-10 are repeated until each triangle has been processed in the manner described above. 
[0105] The result of performing the processing described above is a VRML (or similar format) model of the surface 

40 of subject object 210 and the object support 220, complete with texture coordinates defining image data to be rendered 
onto the model. 

[0106] Referring again to Figure 6, at step S6-14, object/support separator 90 processes the data defining the 3D 
computer model of the surface of subject object 210 and the surface of object support 220 generated by surface 
modeller 70 at step S6-1 0 to remove the object support 220 from the mode!, leaving a 3D computer model of the subject 
45 object 210 alone. 

[0107] In this embodiment, object/support separator 90 performs processing in accordance with user instructions 
input via a user input device 6 to remove interactively the object support 220 from the 3D computer model. 
[0108] Figure 11 shows the processing operations performed by object/support separator 90 at step S6-14. 
[0109] Referring to Figure 11 , at step S11-2, object/support separator 90 causes display processor 110 to display 

50 on display device 4 an image of the 3D computer model of the subject object 210 and object support 220 generated 
by surface modeller 70 rendered with texture data generated by surface texturer 80. In addition, referring to Figure 
12a, object/support separator 90 causes display processor 110 to display a horizontal plane 450 intersecting the dis- 
played 3D computer model, the horizontal plane 450 being movable by the user in a vertical direction (that is, in a 
direction perpendicular to the photographic mat 34) using a user input device 6 such as a mouse. 

55 [0110] At step S11-4, referring to Figures 12a, 12b and 12c, object/support separator 90 moves the plane 450 up 
and/or down in a vertical direction in accordance with user input instructions to a final position defined by the user 
where the plane 450 separates the subject object 210 and object support 220 in the 3D computer model (Figure 12c). 
In addition, object/support separator 90 causes display processor 11 0 to display the 3D computer model of the subject 
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object 210, object support 220 and plane 450 from one or more different viewing positions and directions in accordance 
with user input instructions. This enables the user to view the 3D computer model from different viewpoints and move 
the plane 450 while viewing the 3D computer model from these viewpoints in order to ensure that the final position of 
the plane 450 accurately separates the subject object 21 0 from the object support 220 in the 3D computer model (plane 
5 450 always being constrained to move in a direction perpendicular to the calculated plane of the photographic mat 34, 
which may not be a vertical direction on the display device 4 when the 3D computer model is viewed from different 
viewing positions and directions). 

[0111] At step S11-6, in response to a user input signal indicating that the user has finished moving the plane 450, 
so that the plane is now in a position separating the subject object 21 0 and object support 220, object/support separator 

10 90 stores the coordinates defining the final position of the plane. 

[0112] At step S11-8, object/support separator 90 performs processing to delete the part of the 3D surface model 
generated by surface modeller 70 lying below the final position of the plane stored at step S11 -6 (that is, to delete the 
part of the model lying on the side of the plane 450 towards the calculated plane of the photographic mat 34). 
[0113] Referring to Figure 13, to perform the processing at step S11-8, object/support separator 90 repeats the 

15 processing performed by surface modeller 70 at step S6-1 0, but this time amending the cuboid 400 defining the start 
volume for the voxel carve processing so that the horizontal bottom face 412 is moved to a position corresponding to 
the final position of the plane 450 stored at step S11 -6. In this way, when the voxel carve processing and subsequent 
processing (described previously) is performed, a 3D computer model of the surface of the subject object 210 alone 
is generated, and the model does not include any part of the object support 220. 

20 [0114] Referring again to Figure 11, at step S11-10, object/support separator 90 causes display processor 110 to 
display an image of the 3D surface model generated at step S1 1 -8 rendered with the appropriate texture data generated 
by surface texturer 80 at step S6-12 on display device 4. 

[0115] At step S11-12, object/support separator 90 determines whether a user input signal has been received indi- 
cating that further changes are to be made to the 3D computer model to remove the model of the object support 220. 

25 Steps S11-4 to S11-12 are repeated until it is determined at step S1 1-1 2 that no further changes are necessary. 

[0116] Referring again to Figure 6, at step S6-16, central controller 20 outputs the data defining the 3D computer 
model of the subject object 210 and texture data therefor from output data store 120, for example as data stored on a 
storage device such as disk 122 or as a signal 124 (Figure 1). In addition, or instead, central controller 20 may cause 
display processor 1 1 0 to display an image of the 3D computer model of the subject object 21 0 rendered with the texture 

30 data in accordance with a viewpoint input by a user, for example using a user input device 6. Alternatively, the data 
defining the position and orientation of the camera 230 for each input image generated at step S6-6 and the data 
defining the segmentation of each input image generated at step S6-8 may also be output, for example as data recorded 
on a storage device such as disk 122 or as a signal 124. This data may then be input into a separate processing 
apparatus programmed to perform steps SS-^I 0 to S6-1 4. 

35 [0117] In the first embodiment described above, object/support separator 90 processes the 3D computer model of 
the subject object and object support at step S6-14 to remove the object support therefrom in accordance with user 
input instructions defining the position of a plane 450 to separate the subject object and the object support. 
[0118] However, a 3D computer model of the subject object alone (that is, without the object support) may be gen- 
erated in different ways by object/support separator 90, as will be clear from the further embodiments described below. 

40 

Second Embodiment 

[0119] A second embodiment of the invention will now be described. The components of the second embodiment 
and the processing operations performed thereby are the same as those in the first embodiment, with the exception 
45 that the object support is different and the processing operations performed by the object/support separator 90 are 
different, as will be described below. In addition, in the second embodiment texture data for the underside of subject 
object 21 0 is not generated from the input images by surface texturer 80 (this being inessential because the underside 
of the subject object 210 is of no significant interest to the user in many cases). 

[0120] Referring to Figure 14, in the second embodiment, an object support 460 comprises a cylinder having an 

50 opaque bottom portion 470 and a transparent top portion 480. 

[0121] The diameter of the object support 460 is sized so that the subject object 210 covers the entire top surface 
of the object support 460 when it is placed thereon. In this way, the horizontal surface on which the subject object 210 
sits does not protrude beyond the edges of the base of the subject object 210. Accordingly, there is no horizontal 
surface adjacent the subject object 210 on which shadows can be formed, and therefore no shadows can touch the 

55 subject object 210 in an input image. 

[0122] Because the top portion 480 of the object support 460 is transparent, in each input image stored at step S6-4, 
part of the background screen 228 will be visible between subject object 210 and the opaque bottom portion 470 of 
the object support 460, as shown in Figure 14. In this way, in each input image, the subject object 210 will appear to 
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be separated from the bottom portion 470 of the object support 460. 

[0123] Consequently, in the processing performed by image data segmenter 60 at step S6-8, pixels relating to the 
portion of the background screen 228 between the subject object21 0 and the bottom portion 470 of the object support 
460 are designated as "background" pixels, while the pixels relating to the subject object 210 and the pixels relating 
5 to the bottom portion 470 of the object support 460 are designated as separated regions of "object" pixels. 

[0124] Therefore, in the processing performed by surface modeller 70 at step S6-10, a 3D computer model of the 
subject object 210 is generated and a separate, unconnected 3D computer model of the bottom portion 470 of the 
object support 460 is generated. 

[0125] Accordingly, in the processing performed by object/support separator 90 at step S6-14, the 3D computer 
10 model of the bottom portion 470 of the object support 460 merely needs to be distinguished from the 3D computer 
model of the subject object 210 and deleted. 

[0126] Figure 15 shows the processing operations performed by object/support separator 90 at step S6-14 in the 
second embodiment. 

[0127] Referring to Figure 15, at step S15-2, object/support separator 90 causes display processor 110 to display 
is on display device 4 an image of the 3D computer model of the subject object 21 0 and the 3D computer model of the 
bottom portion 470 of the object support 460 generated by surface modeller 70 rendered with texture data generated 
by surface texturer 80. 

[0128] At step S15-4, object/support separator 90 causes display processor 110 to display a message on display 
device 4 requesting the user to identify the 3D model of the bottom portion 470 of the object support 460. 
20 [0129] At step S1 5-6, in response to a signal from the user input via a user input device 6 such as a mouse, object/ 
support separator 90 deletes the 3D computer model defined in the input signal, thereby leaving the 3D computer 
model of the subject object 210 alone. 

[0130] As a modification to this embodiment, the top portion 480 of the object support 460 may be opaque and have 
the same colour as the background screen 228. This has the same effect as a transparent top portion 480 because, 

25 in each input image, the top portion 480 is indistinguishable from the background screen 228. 

[0131 ] As an alternative to the processing described above with reference to Figure 1 5, object/support separator 90 
may delete the 3D computer model of the bottom portion 470 of the object support 460 automatically (that is, without 
input from the user). More particularly, this is achieved by deleting the 3D computer model which is closest to the 
position of the photographic mat 34 calculated by camera calculator 50 at step S6-6, resulting in the correct 3D.computer 

30 model being deleted every time because the subject object 210 is always placed on the object support 460 and is 
therefore always further away from the photographic mat 34 than the bottom portion 470 of the object support 460. 

Third Embodiment 

35 [0132] A third embodiment of the invention will now be described. The components of the third embodiment and the 
processing operations performed thereby are the same as those in the first embodiment, with the exception that the 
object support is different and the processing operations performed by the object/support separator 90 are different, 
as will be described below. In addition, as in the second embodiment, texture data for the underside of subject object 
210 is not generated in the third embodiment. 

40 [01 33] In the third embodiment, the object support upon which the subject object 21 0 is placed for imaging comprises 
a cylinder, as in the second embodiment, but in the third embodiment, the whole of the cylinder is opaque rather than 
part being opaque and part being transparent as in the second embodiment. 

[0134] Consequently, as in the first embodiment, the 3D computer model generated by surface modeller 70 at step 
S6-1 0 comprises a 3D computer model of the subject object 210 together with the object support. 
45 [0135] At step S6-14 in the third embodiment, object/support separator 90 performs processing to test the cross- 
sectional area of the 3D computer model at different positions in a direction moving away from the position of the 
photographic mat 34 previously calculated by camera calculator 50 at step S6-6 (that is, in an upwards direction from 
the base of the object support to the subject object 210). More particularly, object/support separator 90 monitors the 
cross-sectional area of the 3D computer model in order to detect the position at which a sudden change in the cross- 
so sectional area occurs (this sudden change indicating the boundary between the object support and the subject object 
210 provided that the base of the subject object 210 is narrower or wider than the top surface of the object support). 
Having detected the position at which the cross-sectional area of the 3D computer model changes, object/support 
separator 90 performs processing to re-generate the 3D computer model as described in the first embodiment by 
setting the base plane 41 2 of the initial volume 400 on which the voxel carve processing is performed to be the detected 
55 position at which the cross-sectional area of the 3D computer model changes. 

[0136] In this way, a 3D computer model of the subject object 210 alone is generated without user input. 

[0137] As an alternative to the processing described above, an object support having a substantially uniform colour 

which is substantially different to any colour at the bottom of the subject object 210 may be used and, instead of 
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performing processing to detect a sudden change in the cross-sectional area of the object support, object/support 
separator 90 may perform processing to detect the position of a sudden change in colour of the 3D computer model 
when rendered with texture data generated by surface texturer 80. 

[0138] In the first three embodiments described above, the 3D computer model of the subject object 21 0 and object 
5 support is generated (either as a single 3D computer model as in the first and third embodiments or as two separate 
3D computer models as in the second embodiment) and the 3D computer model relating to the object support is 
subsequently removed. 

[0139] However, instead, surface modeller 70 may perform processing at step S6-10 to generate data defining a 3D 
computer model of the subject object alone, as will be clear from the fourth embodiment described below. 

10 

Fourth Embodiment 

[0140] A fourth embodiment of the invention will now be described. The components of the fourth embodiment and 
the processing operations performed thereby are the same as those in the first embodiment, with the exception that 
15 the height of the object support is known and is stored in processing apparatus 2 (for example the height may be input 
by the user), and object/support separator 90 (and consequently processing step S6-14 performed thereby) is not 
included and instead the processing performed by surface modeller 70 is modified. These differences will be described 
below. 

[0141] More particularly, because the height of the object support is known and stored in processing apparatus 2, 
20 at step S6-10, surface modeller 70 sets the base plane 412 of the initial volume 400 on which to perform voxel carve 
processing to be at a vertical distance away from the position of the photographic mat 34 equal to the stored height of 
the object support. In this way, in the subsequent processing performed by surface modeller 70 at step S6-10, a 3D 
computer model of the subject object 210 alone is generated. 

25 Modifications 

[0142] Many modifications can be made to the embodiments described above within the scope of the claims. 
[0143] For example, modifications can be made to the object support upon which the subject object 21 0 is placed 
for imaging. 

30 [0144] By way of a first example, the object support can be designed to assist the user in selecting an optimum 
position and viewing direction for camera 230 since, as described above with reference to Figures 4a-4e, the imaging 
position and direction of the camera 230 are important to ensure that the subject object 210 appears surrounded by 
the background screen 228 in each input image. More particularly, referring to Figure 1 6, to assist the user in achieving 
the correct imaging position and direction, an object support 490 may be used having a marker 500 thereon so that 

35 the user can select the position and direction of the camera 230 such that the marker 500 appears at the boundary 
between the photographic mat 34 and the background screen 228 - the marker 500 being positioned at a sufficient 
distance away from the top of the object support such that when this alignment occurs, the subject object 210 will 
always be surrounded by the background screen 228. 

[0145] Referring to Figure 17, the object support and photographic mat 34 may be combined by providing an object 
40 support 51 0 on which the pattern of features from the photographic mat 34 is printed (or otherwise marked) at a position 
separated from the top surface of the object support on which the subject object 210 sits. 

[0146] In addition to providing features on the photographic mat 34 for detection in the input images, as in the em- 
bodiments described above, features may also be provided on the object support, and processing may be performed 
by camera calculator 50 to detect the features on the object support in each input image and to match the detected 

45 features between input images. The positions of the detected and matched features on the object support may then 
be used together with the detected positions of the features on the photographic mat 34 to calculate the positions and 
orientations of the input images. This is likely to increase the accuracy of the calculated positions and orientations (and 
hence lead to a more accurate 3D computer model of the subject object) because the features now being used to 
calculate the positions and orientations do not all lie in a single plane (which is the case in the embodiments described 

50 above because ail of the features are on the photographic mat 34) and instead the features on the object support can 
be provided at different heights thereon. As an example of the way in which the detected features on the photographic 
mat 34 and the detected features on the object support may be used to calculate the positions and orientations of the 
input images, processing may be performed based on that described in EP-A-0898245 using the matches between 
features on the photographic mat 34 as "user-identified" matches and the matches between the features on the object 

55 support as "calculated" matches. 

[0147] In addition, the features on the object support may be provided at known relative positions thereon. In this 
way, the positions and orientations of the input images may be calculated with further increased accuracy by using the 
known relative positions to constrain the solution that each calculated position and orientation can take in a conventional 
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manner (that is, by requiring the calculated position and orientation to give imaging conditions which preserve the 
relative positions of the features on the object support). 

[0148] Yet further, the features on the object support may be provided at known relative positions and in addition the 
object support and photographic mat may be arranged to interconnect so that, when connected, the positions of the 

5 features on the object support are known relative to the positions of the features on the photographic mat. More par- 
ticularly, referring to Figure 18, a photographic mat 520 may be used comprising a pattern of features on a slab of 
material having a hole 530 therein. An object support 540 having a plurality of features 560 each with a respective 
different colour and arranged at known relative positions interconnects with the photographic mat 520 by inserting the 
base thereof in the hole 530. The relative positions of the features on the photographic mat 520 and the features 560 

10 on the object support 540 are fixed when the object support 540 is inserted in the hole 530 because a protrusion 550 
is provided on the object support 540 which engages a notch in the hole 530. 

[0149] In the embodiments described above, the calibration object on which the object support and subject object 
210 are placed has the form of a two-dimensional photographic mat 34. However, instead, a three-dimensional cali- 
bration object may be used. For example, referring to Figure 19, a calibration object 570 having the form of a cube 

15 with a pattern of features on the vertical sides thereof may be used. 

[01 50] The object support may be telescopic to allow the height thereof to be changed. 

[0151] The object support may take the form of a spike to be inserted into the subject object 21 0. 

[0152] In all of the embodiments described above, the object support has a flat top surface on which the subject 

object 21 0 sits. However, the subject object may be supported by separate pillars. For example, in the first embodiment, 

20 the horizontal top plate 222 of the object support may be removed and the subject object supported on the legs 224. 
In embodiments in which texture data is to be generated for the underside of the subject object, this arrangement would 
provide a reflection of at least part of the underside in the mirror 1 40 unaffected by distortion resulting from transmission 
of light through the surface on which the subject object is sitting (for example surface 222 in the first embodiment). 
[01 53] In all of the embodiments described above, the features of the pattern on the photographic mat 34 are arranged 

25 around an empty central area in which the object support is placed. However, the features may be provided over a 
photographic mat 34 without a space for the object support, and the object support may be then placed over some of 
the features. This will not affect the processing operations to calculate the position and orientation of each input image 
provided that a minimum of six features are visible in each input image. 

[0154] In the embodiments described above, the object support with the subject object 210 thereon is positioned on 

30 the photographic mat 34 so that it is surrounded by the features in the pattern on the photographic mat. However, the 
object support with the subject object 210 thereon may be positioned on the photographic mat 34 so that it is outside 
the pattern of features (for example in a corner of the photographic mat 34 outside the circle of features). Similarly, the 
object support with the subject object 210 thereon need not be positioned on the photographic mat. For example, if 
camera 230 is fixed, the photographic mat 34 may be placed on a larger sheet of paper, the object support with the 

35 subject object 210 thereon may also be placed on the larger sheet but off the photographic mat 34, and the larger 
sheet may be rotated and/or moved between the recording of images to provide different relative positions and orien- 
tations between on the one hand the camera 230 and on the other hand the photographic mat 34, object support and 
subject object 210. In this way, even though the subject object 210 is not directly over the pattern of features on the 
photographic mat 34, the subject object 210 is still held above the pattern by the object support, and both the subject 

40 object 210 and the pattern will be visible in recorded images. Accordingly, the word "above" used herein should not be 
interpreted to mean that the subject object is directly over the photographic mat in a vertical direction. 
[0155] In all of the embodiments described above, a 3D computer model of the subject object 210 alone is generated. 
However, for some applications, a 3D computer model of the subject object 21 0 together with the object support (or 
part thereof) may be acceptable. In such a case, it is unnecessary to perform processing to remove the 3D computer 

45 model of the object support. 

[0156] In the first embodiment described above, in the processing performed by object/support separator 90 at step 
S6-14, the plane 450 displayed to the user for movement to separate the subject object 210 from the object support 
220 has a width which exceeds the width of any part of the 3D computer model of the subject object 210. Similarly, 
before re-performing voxel carve processing, object/support separator 90 moves the base plane 41 2 of the initial volume 

so 400 to remove the entire bottom portion of the volume 400 beneath the base plane 41 2. This processing is satisfactory 
when, as shown in Figure 5, every part of the subject object 210 is above the top of the object support 220. However, 
a situation may arise in which part of the subject object 210 overhangs the top of the object support 220 so that, using 
the processing described in the first embodiment, it would not be possible to separate the subject object 210 and object 
support 220 using the plane 450. Accordingly, as an alternative, at step S6-1 4, object/support separator 90 may display 

55 a plane 450 having the same shape and cross-sectional area as the object support 220 and, when amending the initial 
volume 400 on which to perform voxel carve processing, instead of moving the base plane 41 2, a hole may be created 
in the volume 400 having a cross-sectional area and shape corresponding to that of the plane 450 displayed to the 
user and extending from the original base plane 412 to a position corresponding to the final position of the plane 450 
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stored at step S1 1 -6. The initial volume 400 may also be amended in a similar way in the third and fourth embodiments 
to take account of situations in which the subject object 21 0 overhangs the top of the object support. 
[0157] As an alternative method of performing processing to remove the 3D computer model of the object support 
from the 3D computer model of the subject object and object support together, a 3D computer model of the object 

5 support may be pre-generated and pre-stored in processing apparatus 2 (for example by performing the processing 
described previously without the subject object 21 0 present when the input images are recorded) and then processing 
may be performed by object/support separator 90 to match the pre-stored model of the object support to part of the 
3D model of the subject and object support together, and to remove the detected matching part. 
[0158] In the embodiments described above, the printer 8 prints photographic mat 34 on a single sheet of paper. 

10 However, mat generator 30 may control printer 8 to print the photographic mat on separate sheets of paper, which can 
then be placed together to form the photographic mat 34. 

[0159] Rather than printing the photographic mat on a recording medium or displaying the photographic mat on 
display panel 1 0, thefeatures of the photographic mat may be marked (for example painted) on a surface such as a floor. 
[0160] In the first embodiment described above, in order to generate texture data for the underside of the subject 

15 object 210, the sheet of paper on which the photographic mat is printed has a hole 130 cut therein and is attached to 
a mirror 140. However, instead, a mirror, or other suitably reflective material, may be provided on top of the sheet of 
paper at a position corresponding to that where the hole 130 would have been cut. Similarly, if the photographic mat 
is displayed on display panel 1 0, a mirror or other reflective material may be placed on the display panel. Alternatively, 
the pattern of features of the photographic mat may be printed, or otherwise marked on a mirror or other reflective 

20 surface. 

[0161] In the first embodiment described above, in the processing performed by object/support separator 90 at step 
S6-14, the 3D computer model generated by surface modeller 70 is amended by moving the base plane 412 of the 
initial volume 400 on which voxel carve processing is to be performed to a position corresponding to the position of 
plane 450 previously moved by the user. In addition, the same technique may be used to amend the 3D computer 

25 model to correct any errors therein. More particularly, referring to Figure 20a, processing apparatus 2 may display an 
image of the 3D computer model together with a plane 600 which can be moved by a user using a user input device 
6 to any position and orientation. In this way, the user can move the plane 600 to a position which separates any 
unwanted part of the computer model, such as part 61 0 in Figure 20a, which protrudes from the wanted part. As in the 
processing described above with reference to Figure 11 , processing apparatus 2 stores the final position of the plane 

30 600 moved by the user and, referring to Figure 20b, amends the initial volume 400 on which voxel carve processing 
is to be performed by setting a plane 620 at a position in the volume 400 corresponding to the position of the plane 
600 defined by the user, and removing the portion 630 from the initial volume 400 excluded by the plane 620. Processing 
apparatus 2 then re-performs the voxel carve processing and, as a result, the unwanted portion 61 0 of the 3D computer 
model is removed because this part lies in the volume 630 excluded from the voxel carve processing. 

35 [0162] In the embodiments described above, at step S6-4, data input by a user defining the intrinsic parameters of 
camera 230 is stored. However, instead, default values may be assumed for some, or all, of the intrinsic camera 
parameters, or processing may be performed to calculate the intrinsic parameter values in a conventional manner, for 
example as described in "Euclidean Reconstruction From Uncalibrated Views" by Hartley in Applications of Invariance 
in Computer Vision, Mundy, Zisserman and Forsyth eds, pages 237-256, Azores 1993. 

40 [0163] In the embodiments described above, image data from an input image relating to the subject object 210 and 
object support is segmented from the image data relating to the photographic mat 34, surface 200 and background 
screen 228 as described above with reference to Figure 7. However, other conventional segmentation methods may 
be used instead. For example, a segmentation method may be used in which a single RGB value representative of 
the colour of the photographic mat 34, surface 200 and background screen 228 is stored and each pixel in an input 

45 image is processed to determine whether the Euclidean distance in RGB space between the RGB background value 
and the RGB pixel value is less than a specified threshold. 

[0164] In the embodiments described above, the processing at steps S6-1 0 and S6-14 to generate data defining a 
3D computer model is carried out using a voxel carving technique. However, other techniques may be used, such as 
a voxel colouring technique for example as described in University of Rochester Computer Sciences Technical Report 
50 Number 680 of January 1 998 entitled "What Do N Photographs Tell Us About 3D Shape?" and University of Rochester 
Computer Sciences Technical Report Number 692 of May 1998 entitled "A Theory of Shape by Space Carving", both 
by Kiriakos N. Kutulakos and Stephen M. Seitz. 

[0165] In the embodiments described above, processing is performed by a computer using processing routines de- 
fined by programming instructions. However, some, or all, of the processing could be performed using hardware. 

55 
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Claims 

I. A method of recording images of a subject object from different positions and orientations and processing the 
recorded image data to generate a three-dimensional computer model of the subject object, comprising: 

5 

supporting the subject object above a calibration object having a predetermined pattern of features using an 
object support; 

recording at different positions and orientations a plurality of images of the subject object supported above 
the calibration object; 

w processing the recorded image data to calculate the position and orientation at which each of at least some 

of the images were recorded; and 

performing processing using the calculated positions and orientations to generate data defining a three-di- 
mensional computer model of at least the subject object. 

15 2. A method according to claim 1 , wherein the images of the subject object supported above the calibration object 
are recorded with a background of a substantially uniform colour behind the subject object such that, in each 
recorded image, the outline of the subject object is surrounded by the background except where the outline touches 
the support. 

20 3. a method according to claim 2, wherein the background is provided by a background screen. 

4. A method according to claim 2 or claim 3, wherein at least the part of the object support adjacent the subject object 
is substantially the same colour as the background, and wherein the processing to generate the three-dimensional 
computer model comprises generating data defining a three-dimensional computer model of the subject object 

25 and a separated three-dimensional computer model of the object support. 

5. A method according to any of claims 1 to 3, wherein at least the part of the object support adjacent the subject 
object is substantially transparent, and wherein the processing to generate the three-dimensional computer model 
comprises generating data defining a three-dimensional computer model of the subject object and a separated 

30 three-dimensional computer model of the object support. 

6. A method according to any preceding claim, wherein the subject object is supported by at least one surface of an 
object support standing on the calibration object, and wherein each surface of the object support supporting the 
subject object does not protrude substantially from beneath the subject object. 

35 

7. A method according to any preceding claim, wherein the object support has calibration features thereon, and 
wherein the processing to calculate the position and orientation at which each of at least some of the images were 
recorded includes detecting calibration features on the object support in image data and using the positions of the 
detected features to calculate the positions and orientations at which the images were recorded. 

40 

8. A method according to claim 7, wherein data defining the relative positions of the calibration features on the object 
support is prestored and used to calculate the positions and orientations at which the images were recorded. 

9. A method according to claim 7 or claim 8, wherein the object support is arranged relative to the calibration object 
45 jn a predetermined configuration, and wherein data defining the positions of the calibration features on the object 

support relative to the positions of the features on the calibration object is prestored and used to calculate the 
positions and orientations at which the images were recorded. 

10. A method according to any preceding claim, wherein the subject object is supported above a reflective surface, 
50 and wherein processing is carried out to generate texture data for the three-dimensional computer model of the 

subject object in dependence upon image data that corresponds to reflections in the reflective surface. 

II. A method according to any preceding claim wherein the calibration object is three-dimensional. 

55 12. A method according to any preceding claim, wherein the object support and the calibration object are formed as 
one, with the subject object being supported thereby above the predetermined pattern of features thereon. 

13. A method of processing image data defining a plurality of images recorded at different positions and orientations 
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-vf a subject object supported by an object support above a calibration object having a predetermined pattern of 
eatures, comprising: 

processing the image data to calculate the positions and orientations at which at least some of the images 
5 were recorded, and 

performing processing using the calculated positions and orientations to generate data defining a three-di- 
mensional computer model of the subject object but not the object support. 

14. A method according to claim 13, wherein the step of calculating the positions and orientations at which at least 
10 some of the images were recorded includes detecting matching features in the image data defining respective 

images corresponding to features on the object support. 

15. A method according to claim 13 or claim 14, wherein the step of generating the data defining the three-dimensional 
computer model comprises: 

15 

processing images to segment image data relating to at least the subject object from background image data; 
and 

processing the segmented image data and the calculated positions and orientations to generate the data 
defining the three-dimensional computer model. 

20 

16. A method of processing image data to generate a three-dimensional computer model, comprising: 

receiving image data defining at least in part a plurality of images of a subject object supported by an object 
support recorded at different relative positions and orientations; 
25 receiving data defining the positions and orientations at which the images were recorded; and 

processing the received data to generate data defining a three-dimensional computer model of the subject 
object but not the object support. 

17. A method according to any of claims 13 to 16, wherein the step of generating data defining the three-dimensional 
30 computer model comprises performing processing using at least one known parameter of the object support to 

generate data defining the three-dimensional computer model of the subject object without generating data defining 
a three-dimensional computer model of the object support. 

18. A method according to claim 17, wherein the known parameter of the object support is the height of the object 
35 support. 

19. A method according to claim 18, wherein the step of generating data defining the three-dimensional computer 
model of the subject object includes: 

40 defining a volume of voxels in a three-dimensional space in dependence upon the known height of the object 

support such that the object support, but not the subject object is excluded from the volume; and 
removing voxels from the volume in dependence upon the image data. 

20. A method according to any of claims 13 to 16, wherein the step of generating data defining the three-dimensional 
45 computer model comprises performing processing to generate at least one three-dimensional computer model of 

the subject object and object support and performing processing to remove the three-dimensional computer model 
of the object support. 

21. A method according to claim 20, wherein the processing to remove the three-dimensional computer model of the 
50 object support is carried out in dependence upon signals input by a user defining the three-dimensional computer 

model to be removed. 

22. A method according to claim 21 , wherein the processing to remove the three-dimensional computer model of the 
object support comprises: 



55 



(a) generating image data for display to a user defining an image of the three-dimensional computer model of 
the subject object and object support together with a plane moveable by the user; 

(b) receiving signals input by the user defining a position of the plane; 
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(c) repeating steps (a) and (b); and 

(d) removing the three-dimensional computer model which lies on a predetermined side of the plane. 

23. A method according to claim 22, wherein: 

5 

the subject object is supported on the top-most surface of an object support standing on the calibration object; 
the plane is generated so as to have an orientation in a substantially horizontal plane; and 
processing is performed to allow the user to move the position of the plane but not to change the orientation 
of the plane. 

10 

24. A method according to claim 22 or claim 23, wherein the plane is generated with the same shape and cross- 
sectional area as the object support. 

25. A method according to any of claims 22 to 24, wherein the step of removing the three-dimensional computer model 
is which lies on the predetermined side of the plane comprises: 

defining a volume of voxels in a three-dimensional space such that a boundary of the volume is at a position 
corresponding to the position of the plane; and 

removing voxels from the volume in dependence upon the image data. 

20 

26. A method according to claim 21 , wherein processing is carried out to generate a three-dimensional computer model 
of the subject object and a separate three-dimensional computer model of the object support, and wherein the 
step of removing the three-dimensional computer model of the object support comprises generating image data 
for display to the user defining at (east one image of the three-dimensional computer models, receiving signals 

25 input by the user defining one of the three-dimensional computer models and removing one of the three-dimen- 

sional computer models in dependence upon the signals input by the user 

27. A method according to claim 20, wherein the processing to remove the three-dimensional computer model of the 
object support includes processing to identify the three-dimensional computer model to remove without signals 

30 input by a user. 

28. A method according to claim 27, wherein processing is carried out to generate a three-dimensional computer model 
of the subject object and a separate three-dimensional computer model of the object support, and wherein process- 
ing is carried out to remove the three-dimensional computer model having a position closest to the position cor- 

35 responding to the position of the calibration object. 

29. A method according to claim 27, wherein processing is carried out to test at least one property of the three-dimen- 
sional computer model at different positions and to remove a part of the three-dimensional computer model in 
dependence upon a position at which the tested property changes. 

40 

30. A method according to claim 29, wherein at least one of the cross-sectional area and colour of the three-dimensional 
computer model is tested. 

31 . A method according to claim 27, wherein data defining a reference three-dimensional computer model of the object 
45 support is prestored, the reference model is compared against the three-dimensional computer model of the subject 

object and the object support to identify the part thereof which corresponds to the object support, and the identified 
part is removed. 

32. A method according to any of claims 1 3 to 31 , wherein the step of generating the data defining the three-dimensional 
50 computer model includes generating texture data using the image data. 

33. A method according to claim 32, wherein the step of generating texture data includes processing the image data 
to identify data corresponding to a reflection of the subject object in a reflective surface, and using the identified 
data to generate texture data for a surface of the three-dimensional computer model. 

55 

34. A method according to claim 16, wherein the received image data comprises image data relating to the subject 
object and object support previously segmented from other image data in the recorded images. 
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35. A method according to any of claims 13 to 34, further comprising the step of generating a signal conveying data 
defining the three-dimensional computer model of the subject object. 

36. A method according to claim 35, further comprising the step of recording the signal either directly or indirectly. 

5 

37. A method of processing images of an object to generate data defining a three-dimensional computer model of the 
object and to amend the three-dimensional computer model, comprising: 

(a) receiving image data defining at least in part a plurality of images of an object recorded at different positions 
10 and orientations; 

(b) receiving data defining the positions and orientations at which the images were recorded; 

(c) processing the received data to generate data defining a three-dimensional computer model of the object; 

(d) generating data for display to a user defining an image of the three-dimensional computer model together 
with a plane moveable by the user; 

15 (e) receiving signals input by the user defining a position and orientation of the plane; 

(f) repeating steps (d) and (e); and 

(g) removing the part of the three-dimensional computer model which lies on one side of the plane. 

38. A method according to claim 37, wherein step (g) comprises processing image data received in step (a) and position 
20 and orientation data received in step (b) to generate data defining a three-dimensional computer model of the 

object in a three-dimensional space having a boundary defined by the plane. 

39. A system for recording images of a subject object from different positions and orientations and for processing the 
recorded image data to generate a three-dimensional computer model of the subject object, comprising: 

25 

a calibration object having a predetermined pattern of features; 

an object support for supporting the subject object higher than the calibration object; 

an imager for recording at different positions and orientations a plurality of images of the subject object sup- 
ported higher than the calibration object; and 
30 an image data processing apparatus, comprising: 

means for processing the recorded image data to calculate the position and orientation at which each of 
at least some of the images were recorded; and 

means for performing processing using the calculated positions and orientations to generate data defining 
35 a three-dimensional computer model of at least the subject object. 

40. A system according to claim 39, further comprising a screen having a substantially uniform colour for placing behind 
the subject object so that images of the subject object supported higher than the calibration object can be recorded 
with the screen behind the subject object such that, in each recorded image : the outline of the subject object is 

40 surrounded by the background except where the outline touches the support. 

41. A system according to claim 40, wherein: 

at least the part of the object support adjacent the subject object when the subject object is placed thereon is 
45 substantially the same colour as the screen; and 

the processing means for generating the three-dimensional computer model comprises means for generating 
data defining a three-dimensional computer model of the subject object and a separated three-dimensional 
computer model of the object support. 

50 42. A system according to claim 39 or claim 40, wherein: 

at least the part of the object support adjacent the subject object when the subject object is placed thereon is 
substantially transparent; and 

the processing means for generating the three-dimensional computer model comprises means for generating 
55 data defining a three-dimensional computer model of the subject object and a separated three-dimensional 

computer model of the object support. 

43. A system according to any of claims 39 to 42, wherein the object support is arranged such that, when the subject 
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object sits thereon, no surface supporting the subject object protrudes substantially from beneath the subject object. 

44. A system according to any of claims 39 to 43, wherein: 

the object support has calibration features thereon; and 

the processing means for calculating the position and orientation at which each of at least some of the images 
were recorded includes means for detecting calibration features on the object support in image data and using 
the positions of the detected features to calculate the positions and orientations at which the images were 
recorded. 

45. A system according to claim 44, wherein the image data processing apparatus includes means for prestoring data 
defining the relative positions of the calibration features on the object support, and the processing means for 
calculating the positions and orientations at which each of at least some of the images were recorded is arranged 
to use the prestored data to calculate the positions and orientations at which the images were recorded. 

46. A system according to claim 44 or claim 45, wherein: 

the object support is arranged to connect to the calibration object in a predetermined configuration; 
the image data processing apparatus includes means for prestoring data defining the positions of the calibra- 
tion features on the object support relative to the positions of the features on the calibration object when the 
object support is connected to the calibration object; and 

the processing means for calculating the positions and orientations at which each of at least some of the 
images were recorded is arranged to use the prestored data to calculate the positions and orientations at 
which the images were recorded. 

47. A system according to any of claims 39 to 46, wherein: 

the calibration object has a reflective surface; and 

the image data processing apparatus includes means for generating texture data for the three-dimensional 
computer model of the subject object in dependence upon image data that corresponds to reflections in the 
reflective surface. 

48. A system according to any of claims 39 to 47, wherein the calibration object is three-dimensional. 

49. A system according to any of claims 39 to 48, wherein the object support and the calibration object are formed as 
one with a surface for supporting the subject object such that, when the subject object sits thereon, the subject 
object is supported higher than, and separated from, the predetermined pattern of features. 

50. Apparatus for processing image data defining a plurality of images recorded at different positions and orientations 
of a subject object supported by an object support higher than a calibration object having a predetermined pattern 
of features, comprising: 

means for processing the image data to calculate the positions and orientations at which at least some of the 
images were recorded, and 

means for performing processing using the calculated positions and orientations to generate data defining a 
three-dimensional computer model of the subject object but not the object support. 

51 . Apparatus according to claim 50, wherein the means for calculating the positions and orientations at which at least 
some of the images were recorded includes means for detecting matching features in the image data defining 
respective images corresponding to features on the object support. 

52. Apparatus according to claim 50 or claim 51 , wherein the means for generating the data defining the three-dimen- 
sional computer model comprises: 

means for processing images to segment image data relating to at least the subject object from background 
image data; and 

means for processing the segmented image data and the calculated positions and orientations to generate 
the data defining the three-dimensional computer model. 
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53. Apparatus for processing image data to generate a three-dimensional computer model, comprising: 

means for receiving image data defining at least in part a plurality of images of a subject object supported by 
an object support recorded at different relative positions and orientations; 

means for receiving data defining the positions and orientations at which the images were recorded; and 
means for processing the received data to generate data defining a three-dimensional computer model of the 
subject object but not the object support. 

54. Apparatus according to any of claims 50 to 53, wherein the means for generating data defining the three-dimen- 
sional computer model comprises means for performing processing using at least one known parameter of the 
object support to generate data defining the three-dimensional computer model of the subject object without gen- 
erating data defining a three-dimensional computer model of the object support. 

55. Apparatus according to claim 54, wherein the known parameter of the object support is the height of the object 
support. 

56. Apparatus according to claim 55, wherein the means for generating data defining the three-dimensional computer 
model of the subject object includes: 

means for defining a volume of voxels in a three-dimensional space with the base plane of the volume set to 
be at a height higher than the calibration object corresponding to the known height of the object support; and 
means for removing voxels from the volume in dependence upon the image data. 

57. Apparatus according to any of claims 50 to 53, wherein the means for generating data defining the three-dimen- 
sional computer model comprises means for performing processing to generate at least one three-dimensional 
computer model of the subject object and object support and means for performing processing to remove the three- 
dimensional computer model of the object support. 

58. Apparatus according to claim 57, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support comprises means for removing the three-dimensional computer model in 
dependence upon signals input by a user defining the three-dimensional computer model to be removed. 

59. Apparatus according to claim 58, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support comprises: 

means for generating image data for display to a user defining an image of the three-dimensional computer 
model of the subject object and object support together with a plane moveable by the user; 
means for receiving signals input by the user defining a position of the plane; and 

means for removing the three-dimensional computer model which lies on a predetermined side of the plane. 

60. Apparatus according to claim 59, wherein the subject object is supported on the top-most surface of an object 
support standing on the calibration object, and the means for performing processing to remove the three-dimen- 
sional computer model of the object support is arranged to operate so that: 

the plane is generated so as to have an orientation in a substantially horizontal plane; and 

processing is performed to allow the user to move the position of the plane but not to change the orientation 

of the plane. 

61. Apparatus according to claim 59 or claim 60, wherein the means for performing processing to remove the three- 
dimensional computer model of the object support is arranged to operate so that the plane is generated with the 
same shape and cross-sectional area as the object support. 

62. Apparatus according to any of claims 59 to 61 , wherein the means for removing the three-dimensional computer 
model which lies on the predetermined side of the plane comprises: 

means for defining a volume of voxels in a three-dimensional space such that a boundary of the volume is at 
a position corresponding to the position of the plane; and 

means for removing voxels from the volume in dependence upon the image data. 
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63. Apparatus according to claim 58, wherein: 

the apparatus is arranged to carry out processing to generate a three-dimensional computer model of the 
subject object and a separate three-dimensional computer model of the object support; and 
5 the means for removing the three-dimensional computer model of the object support comprises: 

means for generating image data for display to the user defining at least one image of the three-dimen- 
sional computer models; 

means for receiving signals input by the user defining one of the three-dimensional computer models; and 
10 means for removing one of the three-dimensional computer models in dependence upon the signals input 

by the user. 

64. Apparatus according to claim 57, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support is arranged to operate to identify the three-dimensional computer model to 

15 remove without signals input by a user, 

65. Apparatus according to claim 64, wherein the apparatus is arranged to carry out processing to generate a three- 
dimensional computer model of the subject object and a separate three-dimensional computer model of the object 
support, and the means for performing processing to remove the three-dimensional computer model of the object 

20 support is arranged to carry out processing to remove the three-dimensional computer model having a position 

closest to the position corresponding to the position of the calibration object. 

66. Apparatus according to claim 64, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support is arranged to carry out processing to test at least one property of the three- 

25 dimensional computer model at different positions and to remove a part of the three-dimensional computer model 

in dependence upon a position at which the tested property changes. 

67. Apparatus according to claim 66, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support is arranged to test at least one of the cross-sectional area and colour of the 

30 three-dimensional computer model to determine the part of the three-dimensional computer model to remove. 

68. Apparatus according to claim 64, wherein the means for performing processing to remove the three-dimensional 
computer model of the object support comprises: 

35 means for prestoring data defining a reference three-dimensional computer model of the object support; 

means for comparing the reference model against the three-dimensional computer model of the subject object 
and the object support to identify the part thereof which corresponds to the object support; and 
means for removing the identified part. 

40 69. Apparatus according to any of claims 50 to 68, wherein the means for generating the data defining the three- 
dimensional computer model includes means for generating texture data using the image data. 

70. Apparatus according to claim 69, wherein the means for generating texture data includes means for processing 
the image data to identify data corresponding to a reflection of the subject object in a reflective surface, and means 

45 for using the identified data to generate texture data for a surface of the three-dimensional computer model. 

71. Apparatus according to claim 53, wherein the received image data comprises image data relating to the subject 
object and object support previously segmented from other image data in the recorded images. 

so 72. Apparatus for processing images of an object to generate data defining a three-dimensional computer model of 
the object and for amending the three-dimensional computer model, comprising: 

means for receiving image data defining at least in part a plurality of images of an object recorded at different 
positions and orientations; 

55 means for receiving data defining the positions and orientations at which the images were recorded; 

means for processing the received data to generate data defining a three-dimensional computer model of the 
object; 

means for generating data for display to a user defining an image of the three-dimensional computer model 
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together with a plane moveable by the user; 

means for receiving signals input by the user defining a position and orientation of the plane; and 

means for removing the part of the three-dimensional computer model which lies on one side of the plane. 

5 73. Apparatus according to claim 72, wherein the means for removing the part of the three-dimensional computer 
model comprises means for processing received image data and received position and orientation data to generate 
data defining a three-dimensional computer model of the object in a three-dimensional space having a boundary 
defined by the plane. 

10 74. A storage device storing instructions for causing a programmable processing apparatus to become operable to 
perform a method as set out in at least one of claims 13 to 38. 

75. A signal conveying instructions for causing a programmable processing apparatus to become operable to perform 
a method as set out in at least one of claims 13 to 38. 
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(54) Image processing apparatus 
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ing the subject object 210 on an object support instead 
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colour of the calibration and the selection of the imaging 
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port 220 having a top surface on which the subject object 
21 0 sits which does not protrude from beneath the base 
of the subject object 210, the user is provided with flex- 
ibility in the selection of lighting conditions. By imaging 
the subject object 210 with a background screen 228 
behind it, the user is provided with flexibility in the se- 
lection of the surface on which the calibration object 34 
is placed for imaging. 
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